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COMMERCIAL INSERTION INTO VIDEO STREAMS BASED 



ON SURROUNDING PROGRAM CONTENT 



The invention relates generally to televisions and other monitors and 



commercials and, more particularly, to a method and apparatus for inserting commercials 
into a data stream based on the content of the surrounding program. 



producer, or sometimes local commercials are inserted by a cable television provider. It is 
known in the prior art to store commercials and to insert the same into a live broadcast 
stream by temporarily replacing the live stream by a playout of blocks from memory. In 
this way, the original commercial blocks in the live broadcast stream can be replaced by 
other commercial blocks. However, in this approach, the inserted block of commercials 
must last equally as long as the block that it replaces. It is also known in the art to delay a 
live television broadcast stream by means of an additional memory to replace the 
commercials in the live broadcast stream with longer length commercials or shorter length 
commercials. The receiver uses a memory not only to delay the program blocks in the live 
broadcast stream but also to create a delayed stream. If a commercial longer than the 
available gap in the live broadcast stream is to be inserted, the resumption of the live 
stream is delayed. However, this delay may be used in a next block of content to be 
replaced to insert a new content block that lasts shorter. In this way, more freedom is 
created to replace content blocks than is possible with the method described above. Co- 
pending U.S. application S/N 10 / (attorney Docket US030439), the contents of 
which are incorporated herein by its reference, discloses methods and devices for inserting 
replacement commercials into a data stream by detecting one or more commercial portions 
of the data stream and replacing the detected commercial portions with either more or less 
replacement commercial portions. 

However, the original and/or replacement commercials are often of little 
interest to the viewer. Accordingly, various schemes have been developed for inserting 
particular commercials based on information associated with the viewer. For example, the 
commercial breaks can be detected using known techniques, and a suitable commercial can 
be aired during the break. The commercial may be obtained from one or more virtual 
commercial channels where only commercials are present. The particular commercial that 
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is played can be selected based on demographic factors such as the viewer's age, gender, 
income and the like. While such approaches can be acceptable, they do not take into 
account the content of the television program in which the commercials are inserted. 

One specific problem currently faced by viewers is where the commercials 
5 are in some way inconsistent with the surrounding program content. For example, where a 
viewer is watching a suspense movie with dark scenes and during a commercial break a 
commercial is shown with a bright contrast. This can cause disorientation for the viewer 
because it takes time for the user's eyes to adjust to the bright commercial. Similarly, the 
viewer can be watching a slow romantic movie that is suddenly interrupted by a fast and/or 

10 loud commercial. This could also cause discomfort to the viewer. Furthermore, a listener 
can be listening to slow romantic audio that is interrupted by loud commercials that ruin 
the mood of the audio. 

Therefore it is an object of the present invention to provide methods and 
interactive systems that overcome these and other disadvantages associated with the prior 

15 art. 

Accordingly, a method for inserting replacement commercials into a data 
stream is provided. The method comprising: receiving the data stream; detecting at least 
one commercial in the data stream; determining a characteristic of program content at least 
prior to the at least one commercial; and replacing the at least one commercial in the data 
20 stream with a replacement commercial which is selected based at least in part on the 
determining. 

The characteristic can be an audio characteristic in the program content at 
least prior to the at least one commercial. 

The audio characteristic can be a speech pattern of the program content at 

25 least prior to the at least one commercial. In which case, the replacing can comprise 
replacing the at least one commercial with a replacement commercial having a speech 
pattern within a predetermined threshold of the speech pattern of the program content at 
least prior to the at least one commercial. The speech pattern can be selected from a group 
consisting of pitch and intonation. 

30 The audio characteristic can be a music characteristic in the program 

content at least prior to the at least one commercial. In which case, the replacing can 
comprise replacing the at least one commercial with a replacement commercial having a 

2 



US030456 



music characteristic within a predetermined threshold of the music characteristic in the 
program content at least prior to the at least one commercial. The music characteristic can 
be selected from a group consisting of beats, tempo, rhythm, and genre. 

The characteristic can be a visual characteristic in the program content at 
5 least prior to the at least one commercial. In which case, the replacing can comprise 
replacing the at least one commercial with a replacement commercial having a visual 
characteristic within a predetermined threshold of the visual characteristic in the program 
content at least prior to the at least one commercial. The visual characteristic can be 
selected from a group consisting of texture, color, compositions, and objects. 

1 0 The characteristic can be one or more electronic program guide (EPG) 

characteristics of the program content at least prior to the at least one commercial. In 
which case, the replacing can comprise replacing the at least one commercial with a 
replacement commercial having one or more EPG characteristics within a predetermined 
threshold of the EPG characteristics in the program content at least prior to the at least one 

1 5 commercial. The one or more EPG characteristics can be a genre. 

The method can further comprise: determining a recommendation for a 
viewer based on at least one of the viewer*s viewing behavior and explicit 
recommendations of the viewer; wherein the replacing fiirther comprises replacing the at 
least one commercial in the data stream with a replacement commercial which also is 

20 selected based at least in part on the recommendation. 

Also provided is a device for inserting replacement commercials into a data 
stream. The device comprising a processor for receiving the data stream, detecting at least 
one commercial in the data stream, determining a characteristic of program content at least 
prior to the at least one commercial, and replacing the at least one commercial in the data 

25 stream with a replacement commercial which is selected based at least in part on the 
characteristic determination. 

Where the data stream is a video data stream, the device can further 
comprise a display operatively connected to the processor for displaying video 
corresponding to the video data stream. Where the data stream is an audio data stream, the 

30 device can further comprise at least one speaker operatively connected to the processor for 
reproducing audio corresponding to the audio data stream. 
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The device can further comprise: a recommender system for determining a 
recommendation for a viewer based on at least one of the viewer's viewing behavior and 
explicit recommendations of the viewer; wherein the processor replaces the at least one 
commercial in the data stream with a replacement commercial which also is selected based 
S at least in part on the recommendation. 

Also provided are a computer program product for carrying out the methods 
of the present invention and a program storage device for the storage of the computer 
program product therein. 

These and other features, aspects, and advantages of the apparatus and 
10 methods of the present invention will become better understood with regard to the 
following description, appended claims, and accompanying drawings where: 

Figure 1 illustrates a schematic of a first device for carrying out the methods 
of the present invention. 

Figure 2 illustrates a schematic of a second device for carrying out the 
1 5 methods of the present invention. 

Although this invention is applicable to numerous and various types of 
devices, it has been found particularly useful in the environment of televisions and other 
monitors for displaying video and/or images. Therefore, without limiting the applicability 
of the invention to televisions and other monitors, the invention will be described in such 
20 environment. However, those skilled in the art will appreciate that the devices of the 

present invention can also reproduce audio, such as radio tuners. Furthermore, although 
the invention is described below with regard to a "broadcast" data stream, those skilled in 
the art will appreciate that the same is discussed by way of example only and not to limit 
the invention thereto. For example, the invention is equally applicable to streaming video 
25 content as well which can be viewed via the Internet using a broadband network. 

Referring now to Figure 1 , there is illustrated a first device for inserting 
commercials into a data stream, the first device being configured as a television 100. The 
television has a display screen 102 such as a CRT, an LCD, or a projection screen. The 
television 100 further has a processor 104 that receives the data stream in the form of a 
30 video content (hereinafter referred to simply as "video") input signal 106. The video input 
signal 106 can be from any source known in the art, such as cable, broadcast television, 
satellite, or an external source such as a tuner, VCR, DVD, or set-top box. The video input 

4 



US030456 



signal 1 06 can be a broadcast signal or streaming video, such as from the Internet. The 
video input signal contains both program content portions and original commercial 
portions. The program portions can be sporting events, television shows, movies and the 
like. The original commercial portions can be one or more advertisements for products 
5 and/or services, public announcements, and the like. The processor 104 is further 

operati vely connected to a storage device 1 08 for storing data, settings, and/or program 
instructions for carrying out the conventional functions of the television 1 00 as well as the 
methods of the present invention. Although shown as a single storage device 108, the 
same may be implemented in several separate storage devices that may be any of many 

10 different types of storage devices known in the art. 

The processor 104 receives the video input signal 106, processes the same, 
as necessary, as is known in the art and outputs a signal 1 10 to the display screen in a 
format compatible with the display screen 102. The display screen 102 displays a video 
portion of the video input signal 106. An audio portion 1 12 of the video input signal 106 is 

15 reproduced on one or more speakers 114 also operatively connected to the processor 104. 
The one or more speakers 114 may be integral with the television 100, as shown in Figure 
1 or separable therefrom. 

Referring now to Figure 2, there is shown a second device for inserting 
commercials into a data stream, the second device being configured as an external source, 

20 such as a set-top box, tuner, computer, DVD, or VCR. The external source is generally 

referred to herein by reference numeral 1 50 and refers generally to any device that supplies 
a video input signal to a display device, such as the television 100. In the configuration of 
Figure 2, the television 100 may be as configured in Figure 1 or it may simply be a monitor 
under the control of a processor 152 contained in the external source 150. Thus, as shown 

25 in Figure 2, the input video signal 106 from the processor 152 of the external source 150 
may be directly input to the display screen 102 or to the display screen via the television 
processor 104. The processor 152 is operatively connected to a storage device 154 which 
may be implemented as one or more separable storage devices. The storage device 154 
includes data and settings as well as program instructions for the normal operation of the 

30 external source and/or television 100 as well as for carrying out the methods of the present 
invention. The external source 150 may also include a recommender 156 for generating a 

m 

user profile based on an implicit and/or explicit recommendation of the viewer's viewing 
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preferences. Generation of recommendations is well known in the art. Implicit 
recommendations are formed from an analysis of the viewer's viewing behavior while 
explicit recommendations are generated from feedback entered by the viewer regarding 
his/her likes and dislikes. The recommendation generated can also use a combination of 
5 the implicit and explicit methods. Although shown separable from the processor 1 52, the 
recommender can be a set of instructions that is carried out by the processor 1 52. 

As will be discussed below, depending upon the configuration of the device, 
the processor 104, 152 receives the data stream 106, detects at least one commercial in the 
data stream 1 06, determines a characteristic of program content at least prior to the 

10 commercial, and replaces the commercial in the data stream with a replacement 

commercial which is selected based at least in part on the characteristic determination. The 
processor 1 04, 1 52 can also replace the commercial in the data stream with a replacement 
commercial which also is selected based at least in part on the recommendation generated 
by the recommender system 156. 

15 The processor 104, 152 can be operatively connected to a remote network 

116, such as through a modem (not shown) under the control of the processor 104, 152. 
The remote network 116 can contain a database of replacement commercials that is 
remotely stored and which may be an Internet web site. The replacement commercials 
may also be stored locally at the storage device 108, 154 which can be periodically updated 

20 from the remote network 116. The replacement commercials may also be appended to the 
data stream 106 and used "on the fly" or stored periodically at the storage device 108, 152. 
As discussed above, the replacement commercials can also be located at another channel in 
the data stream. If the storage device 108, 152 or allocated portion thereof becomes fiill 
with replacement commercials, they can be purged by any methods known in the art such 

25 as "first-in-first-out" (fifo). Alternatively, the replacement commercials can be purged 
according to their dissimilarity with the viewer's user profile as generated by the 
recommendation system 156. 

A method for inserting replacement commercials into a data stream will 
now be discussed with regard to Figures 1 and 2. The method includes receiving the data 

30 stream. As discussed above, the data stream can be a cable transmission, a broadcast 
transmission, a satellite transmission, or streaming data. The processor 104, 152 is 
assumed to have or be operatively connected with any hardware necessary to receive, 
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demultiplex, and format such transmissions for display on the display screen 102 and/or 
reproduction on the speaker 1 14. The processor 104, 152 further detects at least one 
commercial in the data stream 106 and possibly each commercial in the data stream 106. 
Methods for detecting commercials in a data stream are well known in the art. 

The processor 104, 152 next determines a characteristic of program content 
at least prior to the (each) detected commercial. The determination can also be made after 
the (each) commercial; however, such would involve storing the data stream and delaying 
its display and/or reproduction. The characteristic can be any characteristic of the program 
content. After the characteristic(s) are determined, the (each) commercial in the data 
stream is replaced with a replacement commercial that is selected based at least in part on 
the determination. As discussed above, the replacement commercial can be selected fi-om a 
database at the remote network 1 16, at the storage device 108, 154, or from the data stream 
106 itself. The processor 104, 152 is assumed to have or to be operatively connected to 
any hardware necessary to retrieve the replacement commercials from a source and to 
format and insert the same into the data stream or otherwise display and/or reproduce the 
replacement commercial. 

One example of such a characteristic is an audio characteristic in the 
program content, such as a speech pattern in the program content or a music characteristic 
in the program content. In the case of a speech pattern, the replacing of the commercials in 
the data stream 106 can comprise replacing the commercial with a replacement commercial 
having a speech pattern within a predetermined threshold of the speech pattern of the 
program content at least prior to the commercial. Thus, the speech pattern of the 
replacement commercial(s) must be determined or identified and compared with the speech 
pattern of the program content surrounding the commercial to determine a degree of 
similarity. If the speech pattern of the replacement commercial and program content are 
similar, the replacement commercial can be inserted in place of the original commercial. 
Examples of speech patterns are pitch and intonation. Detection of speech patterns in data 
is well known in the art. 

Similarly, in the case of a music characteristic, the replacing of the 
commercials in the data stream 106 with a replacement commercial can comprise replacing 
the commercial with a replacement commercial having a music characteristic within a 
predetermined threshold of the music characteristic in the program content at least prior to 
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the commercial. Thus, one or more of the music characteristics of the replacement 
commercials must be determined or identified and compared with the corresponding music 
characteristics of the program content surrounding the commercial to determine a degree of 
similarity. If the music characteristic(s) of the replacement commercial and program 
5 content are similar, the replacement commercial can be inserted in place of the original 
commercial. Examples of music characteristics are beats, tempo, rhythm, and genre. 
Detection of such music characteristics are well known in the art. 

Another example of a characteristic of the program content can be a visual 
characteristic of the program content at least prior to the (each) commercial. In which 

10 case, the replacing of the original commercial(s) in the data stream with a replacement 
commercial can comprise replacing the original commercial with a replacement 
commercial having a visual characteristic within a predetermined threshold of the visual 
characteristic in the program content at least prior to the at least one commercial. Thus, 
one or more of the visual characteristics of the replacement commercials must be 

1 5 determined or identified and compared with the corresponding visual characteristics of the 
program content surrounding the commercial to determine a degree of similarity. If the 
visual characteristic(s) of the replacement commercial and program content are similar, the 
replacement commercial can be inserted in place of the original commercial. Examples of 
visual characteristics can include texture, color, compositions, and objects. Detection of 

20 such visual characteristics in program content are well known in the art. 

Still another example of a characteristic is one or more electronic program 
guide (EPG) characteristics of the program content at least prior to the at least one 
commercial. In which case the replacing of the original commercial(s) in the data stream 
with a replacement commercial can comprise replacing the at least one commercial with a 

25 replacement commercial having one or more EPG characteristics within a predetermined 
threshold of the EPG characteristics in the program content at least prior to the at least one 
commercial. Thus, one or more of the EPG characteristics of the replacement commercials 
must be determined or identified and compared with the corresponding EPG characteristics 
of the program content surrounding the commercial to determine a degree of similarity. If 

30 the EPG characteristic(s) of the replacement commercial and program content are similar, 
the replacement commercial can be inserted in place of the original commercial. An 
example of an EPG characteristic is genre. 
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In the examples above, the characteristic(s) of the program content is 
compared to corresponding characteristic(s) of the replacement commercials to determine 
which, if any, of the replacement commercials can be inserted into an original commercial 
block in the data stream. The comparison can determine the characteristics of the 
5 replacement commercials from an analysis of the same or such information can be supplied 
with the replacement commercials, such as by a look-up table or appended to the 
replacement commercial itself 

The methods described above can also utilize a recommendation in 
determining a suitable replacement commercial. The recommendation can be used in 

1 0 combination with one or more characteristics of the program portion. Thus, the method 
can include determining a recommendation for a viewer. As is known in the art, the 
determination can be based on the viewer's viewing behavior and/or explicit 
recommendations from the viewer. In which case, the replacing of the original 
commercial(s) in the data stream with a replacement commercial can comprise replacing 

1 5 the commercial(s) in the data stream with a replacement commercial that also is selected 
based at least in part on the recommendation. The generation of recommendations to build 
a user profile based on implicit and/or explicit recommendations is well known in the art. 
Furthermore, it is known in the art to replace commercials in a data stream based on user 
preferences, such as that disclosed in World Patent Application WO 01/33848 A 1, the 

20 contents of which are incorporated herein by its reference. 

Of course, any combination of the above examples or any other 
characteristics can be used in determining a replacement commercial for insertion in the 
data stream. Furthermore, each of the characteristics and/or the recommendation can be 
assigned a different weighing factor so that some characteristics and/or the 

25 recommendation can be weighed more or less than others. The weighing factors can be 
assigned by the viewer through a suitable user interface. 

The viewer can decide that he/she wants to view all of the original 
commercials in the data stream and does not want to replace any of the commercials in the 
data stream, such as during a broadcast of the super bowl. Thus, the viewer can toggle 

30 between settings that enable and disable the methods of the present invention, such as 
through a suitable user interface. 
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The methods of the present invention are particularly suited to be carried out 
by a computer software program, such computer software program preferably containing 
modules corresponding to the individual steps of the methods. Such software can of course 
be embodied in a computer-readable medium, such as an integrated chip or a peripheral 
device. 

While there has been shown and described what is considered to be 
preferred embodiments of the invention, it will, of course, be understood that various 
modifications and changes in form or detail could readily be made without departing from 
the spirit of the invention. It is therefore intended that the invention be not limited to the 
exact forms described and illustrated, but should be constructed to cover all modifications 
that may fall within the scope of the appended claims. 
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